1. Field of the Invention
The invention relates generally to tools by means of which users of an interactive environment may interact with the environment and more specifically to tools for interactively manipulating objects belonging to a virtual environment.
2. Description of Related Art
With the continuing increase in processor power and memory density and decrease in the cost of processors and displays, virtual reality systems, in which computers are used to generate displays that appear to the user to be three-dimensional, have become more than laboratory curiosities. Such displays are termed in the following virtual environments. A natural area of application for virtual environments would seem to be design: most objects are three-dimensional, and a computer-aided design (CAD) system that creates a virtual environment should be able not only to provide the designer with a three-dimensional representation of the object being designed, but should also be able to permit the designer to interact directly with the virtual environment in much the same fashion as a two-dimensional CAD system permits the designer to directly interact with its two dimensional display.
One approach to using virtual environments in CAD systems has been the virtual table, a workbench-like back-projection system. Stereoscopic rendering allows representations of objects being worked on by the designer to appear as if they were floating in space above the table, permitting the designer to grab them, move them, and otherwise manipulate them in the virtual reality produced by the table. The virtual table seems perfectly suited as a virtual environment for interactive product design. The designer can either start with 2D sketches on the tabletop and then successively extrude them into 3D space or work directly in 3D from the beginning. The mere size of the table allows for intuitive, gesture-based interaction of the user's hand in the space above the virtual environment that appears on the tabletop. Thus, the virtual table—as an interactive media—combines the strengths of conventional drawing boards, current CAD systems and interactive virtual environments; hence, the virtual table has the potential to make the CAD workplace into a 3D workspace.
Unfortunately, in most interactive virtual reality systems, the user interacts with the virtual environment by means of obtrusive devices such as data gloves and tethered tracking systems. Current examples of user interfaces for interacting with virtual environments include the following: Chu et al. are working on a multi-modal VR-CAD system. See C.-C. P Chu, et al., “Multi-sensory user-interface for a virtual-reality-based computer-aided design system”, Computer-Aided Design, vol. 29, no. 10., pp. 709-725, 1997. This application can be used at a virtual table driven by data gloves. Forsberg et al. have extended their SKETCH system towards 3D and use two magnetic trackers at a virtual table for object transformation with the non-dominant hand and 3D sketching with the dominant hand. See R C. Zeleznik, K. P. Herndon and J. F. Hughes. “SKETCH: An Interface for Sketching 3D Scenes”, Computer Graphics (Siggraph'96 Proceedings), pages 163-170, August 1996, and A. Forsberg, J. LaViola, L. Markosian, R Zeleznik. “Seamless Interaction in Virtual Reality”, IEEE Computer Graphics and Applications, 17(6):6-9, November/December 1997. Fröhlich et al. at Stanford University have developed an architectural application that has assembly/disassembly features and snapping capabilities, as presented in B. Fröhlich, and M. Fischer: “Erstellen von Produktionsmodellen und Produktionsplanung auf der Responsive Workbench”. Wirtschaftsfaktor Virtual Reality: Planen, Bauen und Marketing mit VR, Congress 98, ARCiTEC, Graz, Austria, Mar. 5-6, 1998, CD-ROM Proceedings. URL: http://www-graphics.stanford.edu/projects/RWB/roof/. These research efforts show that there is a trend from using a virtual table as a presentation device towards exploiting it for interactive and collaborative applications. See for example L. D. Cutler, B. Fröhlich, and P. Hanrahan: Two-Handed Direct Manipulation on the Responsive Workbench, 1997 Symposium on Interactive 3D Graphics, pp. 107-114, ACM Press, 1997
Butterworth's 3DM (J. Butterworth, A. Davidson, S. Hench, and T. M. Olano: 3DM—A three dimensional modeler using a head-mounted display. Communications of the ACM, 1992, pp. 55-62.) was one of the first modeling tools using virtual reality technology. The CRIMP system from the University of North Carolina is also an immersive modeling system. It is presented in M. R. Mine. “Working in a virtual world: interaction techniques used in the Chapel Hill immersive modelling program”. Technical Report 1996-029, 1996. 3-Draw by Sachs, Roberts and Stoops (E. Sachs, A. Roberts, and D. Stoops. “3-Draw: A tool for designing 3D shapes”. IEEE Computer Graphics and Applications, pages 18-25, Nov. 1991) allows creation of 3D line sketches in a two-handed fashion by drawing lines with a pen on a tablet while moving the tablet through space. Liang and Green used a flying 3D device in their highly interactive modeler JDCAD, discussed in J. D. Liang and M. Green. “JDCAD: A highly interactive 3D modelling system”. Computers and Graphics, 18(4), pages 499-506, 1994. THRED (C. Shaw and M. Green: “Two-Handed Polygonal Surface Design”, UIST 94, pp. 205-213, ACM Press, 1994) is a tool for two-handed polygonal surface modeling. Stork and Maidhof introduced 3D interaction techniques for modeling precisely with 3D input devices. See A. Stork and M. Maidhof “Efficient and Precise Solid Modelling Using 3D input Devices”. Proceedings of ACM Symposium on Solid Modelling and Applications, pp. 181-194, May 14-16, Atlanta, Ga., 1997.
In spite of all this research, virtual tables and other virtual reality systems are typically used only for design review, apparently because the techniques used to interact with the virtual environment are non-intuitive, difficult to learn, and incompatible with standard ways of doing design work.
The most promising techniques for overcoming these interface difficulties involve the use of pen and pad interfaces. These interfaces consist of an opaque planar surface (the pad) that is held in the subordinate hand and a pointing device (typically a pen) that is held in the dominant hand. The virtual reality system projects images on the pad's surface and the pen is used to manipulate the images and thereby to manipulate the virtual environment A paper by Stoakley, et al, “Virtual Reality on a WIM: Interactive Worlds in Miniature”, CHI '95 Mosaic of Creativity, Denver, Colo., 1995, pp. 265-272, describes the use of a pad to display a miniature representation of the virtual environment By turning the pad, the user can look at the miniature representation from a point of view different from that from which the user is looking at the full-sized virtual environment. The user can also us a pointing device to select and manipulate items on either the miniature representation or the full-sized virtual environment, with manipulations in the one being reflected in the other. If the user's position in the full-sized virtual environment is recorded on the miniature representation, the miniature representation serves as a map. Finally, the miniature representation may work as a “magic lens” to show aspects of the virtual environment that are not visible in the full-sized version.
Angus, et al., disclose the use of a virtual tool called a paddle to interact with a virtual environment in their paper, Ian G. Angus, et al. “Embedding the 2D Interaction Metaphor in a Real 3D Virtual Environment, Proceedings of IS&T/SPIE's Electronic Imaging Symposium, 1995. The paddle is hand-held and has three real buttons. The virtual environment knows the position of the paddle and of a pointing device, and projects 2-D representations onto the paddle's opaque surface. The user can then employ the buttons and the pointing device to interact with the display on the paddle. One use of the paddle is to give the user of the virtual environment a 2-D map of the environment with the user's current position in the environment marked on the map. The user employs the map to locate him or herself within the virtual environment and to navigate to other locations in the environment.
Zsolt Szalavári and Michael Gervautz' paper, “The Personal Interaction Panel—a Two Handed Interface for Augmented Reality”, Eurographics '97, the Eurographics Association, vol. 16, No.3, discloses an opaque pad and pen where the pad is used as a display and the pen is used to select items on the pad and in the virtual environment. When the pen is used to select an item in the virtual environment, the item appears on the pad. The manner in which it is selected by the pen determines how it appears on the pad. The items on the pad may include objects to be added to the virtual environment and tools for manipulating objects in the environment. To add an object on the pad to the environment, the user employs the pen to drag the object from the pad to the desired position in the environment. To employ a tool on the virtual pad, the user employs the pen to select the object and then employs the pen to operate the tool. Among the tools disclosed are scaling tools, rotation tools, cut and paste tools, magic lenses, and coloring tools.
As promising as the pen and pad interfaces are, they still have a number of difficulties and limitations. These difficulties and limitations stem in large part from the fact that the pad and pen are not really part of the virtual environment:                a separate projection arrangement is necessary to project images onto the pad;        the pad and pen obscure the parts of the virtual environment that are behind them;        because the pad is opaque, it has been generally used only to control or display information about the virtual environment, not to interact directly with it. Thus, the most typical use of an opaque pad is as an analogue to a menu or dialog box in a two-dimensional GUI.        
What is needed in general is tools for working with virtual environments that are themselves part of the virtual environment. In the particular area of pad and pen tools, what is needed is tools where the pad and pen are part of the virtual environment, where they do not obscure the virtual environment, and therefore are easily implemented in an environment such as a virtual table which only employs back projection to create the virtual environment.
Further needs in working with virtual environments are techniques for indirectly viewing portions of the virtual environment that are not otherwise accessible and techniques for using objects created by the virtual environment in conjunction with real objects.